AARoads Forum

Please login or register.

Login with username, password and session length

Author Topic: Caltrans Website Rework  (Read 618 times)

cahwyguy

  • *
  • Offline Offline

  • Posts: 352
  • California Highway Guy

  • Age: 59
  • Location: Northridge, CA
  • Last Login: Today at 02:19:50 PM
    • California Highways
Caltrans Website Rework
« on: July 06, 2019, 08:07:02 PM »

I'm working on the highway pages this weekend, and I'm discovering that a lot of links to Caltrans documents -- bridge lots, EIRs, lists of projects approved by the CTC -- that were linked in articles or project pages have disappeared. They may being reworked, but that's poor form. Where I'm bumping into things, I'm tweeting to Caltrans so they can fix them (hopefully). I wanted to let others who might depend on Caltrans websites know, and encourage you to keep Caltrans on their toes and to keep the information out there.

Daniel
Logged
Daniel - California Highway Guy ● Highway Site: http://www.cahighways.org/ ●  Blog: http://blog.cahighways.org/ ● Follow California Highways on Facebook: https://www.facebook.com/cahighways

myosh_tino

  • *
  • Offline Offline

  • Posts: 2712
  • Silicon Valley Roadgeek

  • Age: 45
  • Location: Cupertino, CA
  • Last Login: July 16, 2019, 12:59:49 PM
    • Silicon Valley Roads @ Markyville.com
Re: Caltrans Website Rework
« Reply #1 on: July 09, 2019, 01:07:03 AM »

There are a ton of "dead" links on the redesigned website.  Just about all of the links in the Traffic Ops section are currently broken and a quick sampling of the project documents links on the District 4 site are dead too.  With that said, a rework of this magnitude is no small undertaking and while I believe these documents (sign specs, project documents, etc) will reappear on the site, it's probably going to take some time.
Logged
Quote from: golden eagle
If I owned a dam and decided to donate it to charity, would I be giving a dam? I'm sure that might be a first because no one really gives a dam.

cahwyguy

  • *
  • Offline Offline

  • Posts: 352
  • California Highway Guy

  • Age: 59
  • Location: Northridge, CA
  • Last Login: Today at 02:19:50 PM
    • California Highways
Re: Caltrans Website Rework
« Reply #2 on: July 09, 2019, 10:18:07 AM »

We should probably keep tweeting and reporting the broken links.
Logged
Daniel - California Highway Guy ● Highway Site: http://www.cahighways.org/ ●  Blog: http://blog.cahighways.org/ ● Follow California Highways on Facebook: https://www.facebook.com/cahighways

Techknow

  • *
  • Offline Offline

  • Posts: 86
  • Location: San Francisco, CA
  • Last Login: Today at 11:01:46 AM
Re: Caltrans Website Rework
« Reply #3 on: July 09, 2019, 01:10:08 PM »

Around this time, Caltrans also seem to have removed most of the videos in their YouTube accounts (CaltransVideo and CaltransD7, perhaps more). I don't know if they're deleted, I know some of them are simply unlisted but they show up in the account's playlists.
Logged

nexus73

  • *
  • Offline Offline

  • Posts: 1696
  • Age: 63
  • Location: Coos Bay OR
  • Last Login: Today at 12:35:50 AM
Re: Caltrans Website Rework
« Reply #4 on: July 09, 2019, 03:02:30 PM »

Your tax dollars NOT at work...LOL!

Rick
Logged
US 101 is THE backbone of the Pacific coast from Bandon OR to Willets CA.  Industry, tourism and local traffic would be gone or severely crippled without it being in functioning condition in BOTH states.

ClassicHasClass

  • *
  • Offline Offline

  • Posts: 157
  • 0-60 in 59.999997 years

  • Location: sunny So Cal
  • Last Login: July 21, 2019, 11:39:39 PM
    • Floodgap Roadgap
Re: Caltrans Website Rework
« Reply #5 on: July 10, 2019, 02:45:30 PM »

Yes, it's a great look for them after the gas tax went up again.  :rolleyes:
Logged

fungus

  • *
  • Offline Offline

  • Posts: 10
  • Location: CA
  • Last Login: July 14, 2019, 05:05:01 PM
Re: Caltrans Website Rework
« Reply #6 on: July 11, 2019, 02:15:16 AM »

It's related to AB 434 compliance. The environmental documents were all deleted too. https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=201720180AB434
Logged

cahwyguy

  • *
  • Offline Offline

  • Posts: 352
  • California Highway Guy

  • Age: 59
  • Location: Northridge, CA
  • Last Login: Today at 02:19:50 PM
    • California Highways
Re: Caltrans Website Rework
« Reply #7 on: July 11, 2019, 10:52:27 AM »

Interesting. You can be compliant and still make documents available, so hopefully those documents will return (as that transparancy, I recall, is required by other laws). We just need to keep pushing to bring the documents back.
Logged
Daniel - California Highway Guy ● Highway Site: http://www.cahighways.org/ ●  Blog: http://blog.cahighways.org/ ● Follow California Highways on Facebook: https://www.facebook.com/cahighways

J N Winkler

  • *
  • Offline Offline

  • Posts: 6289
  • Location: Wichita, Kansas/Oxford, Great Britain
  • Last Login: Today at 04:08:36 PM
Re: Caltrans Website Rework
« Reply #8 on: July 11, 2019, 03:25:17 PM »

I didn't say anything here at the time since I don't customarily post about plans collection issues, but back in February I had to do a bottom-up redesign of the script I use to scrape plans and proposals for Caltrans advertised projects.  From about 2000 until (I think) late 2018/early 2019, Caltrans hosted the documentation for advertised contracts under a folder hierarchy on its main server and allowed directory navigation:

http://www.dot.ca.gov/hq/esc/oe/project_ads_addenda/

Each district had its own folder (zero-filled to two places) for its contracts, with subdirectories under each contract folder for plans, specs, addenda, and additional information (mainly CAD files and "informational handouts" that typically include geotechnical data).

Some time ago--I forget exactly when--Caltrans added language to the Office Engineer webpages indicating that a Caltrans account would be necessary to download plans, but did not start enforcing the requirement at that point.  Last winter they rejiggered the contract advertisement pages so that the /project_ads_addenda/ folders can no longer be navigated.  The folder structure has been preserved to an extent, but documents can now be downloaded only by using the partial directory path (scraped from webpages) as an URL argument to a webpage that will return a document only if a login cookie accompanies the HTTP request.

I am not happy about this change, but I have had to recognize that best-practice recommendations for Web design have advised against allowing directory navigation on a HTTP server for several years now, so servers that allow it have been getting rarer and rarer.
Logged
"It is necessary to spend a hundred lire now to save a thousand lire later."--Piero Puricelli, explaining the need for a first-class road system to Benito Mussolini

mrsman

  • *
  • Offline Offline

  • Posts: 2323
  • Age: 43
  • Location: Silver Spring, MD
  • Last Login: Today at 08:12:07 AM
Re: Caltrans Website Rework
« Reply #9 on: July 12, 2019, 09:02:02 AM »

I didn't say anything here at the time since I don't customarily post about plans collection issues, but back in February I had to do a bottom-up redesign of the script I use to scrape plans and proposals for Caltrans advertised projects.  From about 2000 until (I think) late 2018/early 2019, Caltrans hosted the documentation for advertised contracts under a folder hierarchy on its main server and allowed directory navigation:

http://www.dot.ca.gov/hq/esc/oe/project_ads_addenda/

Each district had its own folder (zero-filled to two places) for its contracts, with subdirectories under each contract folder for plans, specs, addenda, and additional information (mainly CAD files and "informational handouts" that typically include geotechnical data).

Some time ago--I forget exactly when--Caltrans added language to the Office Engineer webpages indicating that a Caltrans account would be necessary to download plans, but did not start enforcing the requirement at that point.  Last winter they rejiggered the contract advertisement pages so that the /project_ads_addenda/ folders can no longer be navigated.  The folder structure has been preserved to an extent, but documents can now be downloaded only by using the partial directory path (scraped from webpages) as an URL argument to a webpage that will return a document only if a login cookie accompanies the HTTP request.

I am not happy about this change, but I have had to recognize that best-practice recommendations for Web design have advised against allowing directory navigation on a HTTP server for several years now, so servers that allow it have been getting rarer and rarer.

Does this mean that you can no longer get the documents, or that it is a bit more cumbersome to do so?
Logged

cahwyguy

  • *
  • Offline Offline

  • Posts: 352
  • California Highway Guy

  • Age: 59
  • Location: Northridge, CA
  • Last Login: Today at 02:19:50 PM
    • California Highways
Re: Caltrans Website Rework
« Reply #10 on: July 12, 2019, 11:21:47 AM »

I'll note there's a similar directory approach for the underlying agenda items for the CTC minutes. I haven't done that part of my updates yet, so I have no ideas what horrors will be involved. For a while, they were embedding all the links in a gigantic PDF, which was a royal pain. But I'll find out in the next week or two when I finally get to doing the CTC minutes from April, May, and June. It looks like the CTC liaison page moved to https://dot.ca.gov/programs/transportation-programming/office-of-ctc-liaison-octcl , but the timed agenda page on that link isn't working. The CTC has a page at https://catc.ca.gov/ , and it does have a meetings and events page, but there's a runtime error on that page so you can't get any information. Time to go report it on Twitter.
Logged
Daniel - California Highway Guy ● Highway Site: http://www.cahighways.org/ ●  Blog: http://blog.cahighways.org/ ● Follow California Highways on Facebook: https://www.facebook.com/cahighways

J N Winkler

  • *
  • Offline Offline

  • Posts: 6289
  • Location: Wichita, Kansas/Oxford, Great Britain
  • Last Login: Today at 04:08:36 PM
Re: Caltrans Website Rework
« Reply #11 on: July 12, 2019, 12:13:44 PM »

Does this mean that you can no longer get the documents, or that it is a bit more cumbersome to do so?

I still get the documents.  These changes forced me to sign up for a Caltrans account, and once I log in (which I can automate), everything downloads satisfactorily.  And because the URL arguments still preserve directory paths below contract folder level, I can still perform blockout, i.e., eliminating the URLs for previously downloaded documents from the download list so that only "fresh" stuff comes in each time the script is run.

What I have lost the ability to do is the following:

*  Go to a given district folder and navigate all the contract folders underneath, including folders for contracts that I missed when they were advertised (for a while, Caltrans was using the /project_ads_addenda/ hierarchy as a letting archive and re-uploading contracts that had been deleted from the server years earlier)

*  Go to a given contract folder and download all the documents listed, including "blind" ones that are not linked to from the advertising pages (because directory navigation is no longer possible, every document that is downloaded must have a corresponding partial URL path that can be scraped from a webpage)

I'll note there's a similar directory approach for the underlying agenda items for the CTC minutes. I haven't done that part of my updates yet, so I have no ideas what horrors will be involved. For a while, they were embedding all the links in a gigantic PDF, which was a royal pain. But I'll find out in the next week or two when I finally get to doing the CTC minutes from April, May, and June. It looks like the CTC liaison page moved to https://dot.ca.gov/programs/transportation-programming/office-of-ctc-liaison-octcl , but the timed agenda page on that link isn't working. The CTC has a page at https://catc.ca.gov/ , and it does have a meetings and events page, but there's a runtime error on that page so you can't get any information. Time to go report it on Twitter.

At one time everything was navigable--EIRs, sign specs, . . . .  What a loss!

Regarding the big PDF, I suspect you may already be aware of this approach, but in a similar situation I would look to do PDF to plain text conversion (pdftotext command, IIRC part of the pdftools distribution, comes to mind) to try to grab the URLs using some combination of findstr and a Windows port of sed (if you are on Windows).  I have to do this with Indiana DOT pay item listings to identify which contracts have signing, and with Wisconsin DOT to match letting call numbers (typically of the format YYYYMMDDCCC) with eight-digit project numbers (which I use for filing signing sheets).  It works reasonably well, though there are wrinkles, such as dealing with random spaces that end up inside words as part of the conversion process.  I have not tried documents where the content I want is buried in the PDF code rather than being displayed explicitly as text--AIUI, extraction is still conceptually possible but involves decompressing PDF streams, which can be a hassle.  Pdftk has been my go-to for years, but I use a Windows port of GhostScript (useful for getting rid of unwanted features associated with PDF versions higher than 1.4) and I have found mutools (free) to be much more rapid for some tasks.

Of course, this all relies on getting hold of the documents in the first place, so I thank you for keeping after Caltrans on Twitter.
Logged
"It is necessary to spend a hundred lire now to save a thousand lire later."--Piero Puricelli, explaining the need for a first-class road system to Benito Mussolini

fungus

  • *
  • Offline Offline

  • Posts: 10
  • Location: CA
  • Last Login: July 14, 2019, 05:05:01 PM
Re: Caltrans Website Rework
« Reply #12 on: July 12, 2019, 01:26:03 PM »

If Caltrans puts the construction plans behind passwords that eliminates the need to be AB 434 compliant. But EIRs have plans and technical documents and making a PDF W3 AA compliant could require alt text descriptions on basically every image, and OCRing the scanned in versions of public comment letters (and possibly retyping handwritten comments). https://www.w3.org/TR/WCAG20-TECHS/pdf#PDF7
Logged

cahwyguy

  • *
  • Offline Offline

  • Posts: 352
  • California Highway Guy

  • Age: 59
  • Location: Northridge, CA
  • Last Login: Today at 02:19:50 PM
    • California Highways
Re: Caltrans Website Rework
« Reply #13 on: July 12, 2019, 01:40:35 PM »

I'll note that I heard back from the CTC on this, and they noted: "As you can imagine the lift was very heavy getting everything remediated to acceptable ADA level. We sit under Caltrans web team and they and their vendors are working feverishly to complete the task for us and for themselves in addition to a couple other departments. We were told yesterday that 93% of our documents are remediated and the rest should done soon. They are currently working to link those items ."

Given what fungus noted, they could have quite a lift, given that many of the EIRs and public letters in the CTC minutes are scanned, and the PDF to character transition (I know because I often cut and paste) is not something that makes text to speech work well. Especially on older PDFs, that could be quite a bit of work. We often forget how hard accessibility is. I had a co-worker many many years ago at System Development Corporation who was a blind programmer, and I saw first hand the difficulties of screen to speech. It is much worse now in our very visual computer system world. (PS: She's the person who is behind the LA Radio Reading Service, http://www.larrs.org/ , a great organization)

Logged
Daniel - California Highway Guy ● Highway Site: http://www.cahighways.org/ ●  Blog: http://blog.cahighways.org/ ● Follow California Highways on Facebook: https://www.facebook.com/cahighways

mrsman

  • *
  • Offline Offline

  • Posts: 2323
  • Age: 43
  • Location: Silver Spring, MD
  • Last Login: Today at 08:12:07 AM
Re: Caltrans Website Rework
« Reply #14 on: July 12, 2019, 01:53:46 PM »

If Caltrans puts the construction plans behind passwords that eliminates the need to be AB 434 compliant. But EIRs have plans and technical documents and making a PDF W3 AA compliant could require alt text descriptions on basically every image, and OCRing the scanned in versions of public comment letters (and possibly retyping handwritten comments). https://www.w3.org/TR/WCAG20-TECHS/pdf#PDF7

It is a shame that something that had good intentions has such bad consequences.  It is commendable to have good intentions to make all documentation accessible to those with disabilities, but it is highly impractical to be able to describe every technical detail so that blind people can understand it.  Unfortunately, it has been a long time since common sense prevailed in California.

The written comments should be made accessible though.  Even if it is laborious, it does provide information that would be useful to all, even the blind.  IMO, it shouldn't be too hard to hire readers to read the text alone - the same way they do for books on tape.

This whole saga is also reminiscent of throwing the baby out with the bath water.  Because the current documentation isn't compliant, they simply remove it all from public consumption.  It would have made a whole lot more sense to grandfather existing documentation and only put in the requirements for new letters.

AFAIK, the federal ADA requires new transit facilities to be accessible.  This generally means elevators for those who are wheelchair bound, but other improvements as well for other disabilities.  NYC has a lot of old subway stations without elevators.  The old stations have been grandfathered in.  But certain levels of improvements would require that they update the station for ADA purposes as well.  So there is a great financial disincentive to making necessary improvements.

More about this can be gleamed from this article:

https://www.reuters.com/article/us-new-york-subway-lawsuit/disability-rights-advocates-win-ruling-over-lack-of-nyc-subway-elevators-idUSKCN1QN2OC

Quote
NEW YORK (Reuters) - New York Cityís transit authority violated federal disability law when it replaced a subway stationís stairs without installing an elevator, a federal judge has ruled, a decision that could require new elevators in future station renovations.

The ruling, issued on Tuesday by U.S. District Judge Edgardo Ramos in Manhattan, came as part of a lawsuit against the Metropolitan Transit Authority (MTA) filed in 2016 by disability rights groups, joined last year by the Justice Department.

Ramos found that when the MTA renovates a station in a way that affects its usability, such as by replacing stairs, the federal Americans with Disability Act requires it to install an elevator unless it is technically infeasible, regardless of cost.
Logged

cahwyguy

  • *
  • Offline Offline

  • Posts: 352
  • California Highway Guy

  • Age: 59
  • Location: Northridge, CA
  • Last Login: Today at 02:19:50 PM
    • California Highways
Re: Caltrans Website Rework
« Reply #15 on: July 15, 2019, 10:42:48 PM »

I found a contact at the CTC who provided me with the agenda item background for May and June. One item was one the website rework. This is from that item. Note item 3. What this means is that we should be requesting items more than 3 years old that we have identified (I already have a request in for the bridge logs). We should consider, either on my site or on AAroad or other sites, as appropriate, hosting these items as a public service as the state is not able to.

The Commissionís Website Accessibility Project includes three primary tasks that must be completed before the Departmentís Chief Information Officer will certify compliance by July 1, 2019. These tasks are as follows:

  • The Commissionís website must be redesigned and rebuilt using the most recently revised version of the State of Californiaís Website Template to ensure the underlying infrastructure is compliant with accessibility standards.
  • All content migrated to the Commissionís newly redesigned website (documents, images, graphics, videos, etc.) must be converted to meet the accessibility criteria established in AB 434. This complex conversion process is commonly referred to as document remediation.
  • The Commission must remove content older than three years from its website, and save it internally to be provided upon request.

Commission staff met with California State Transportation Agency (Agency) and Department staff on April 3, 2019, to discuss website compliance issues. At that time, Commission staff learned from Agency and the Department that documents more than three years old should be removed from the Commissionís website since a) the documents are not or might not be compliant with the new standards, and (b) the cost to comply with legal accessibility requirements outweighs the public benefit of leaving older documents on the website.

The Commission staff appreciates that the Department is remediating the Commissionís documents for the last three years to ensure the Commission is in compliance by July 1, 2019. Since the Department Chief Information Officerís signature is required for certification in addition to the signature of the Commissionís Executive Director, and given the cost of outsourcing remediation of all documents currently on the Commissionís website to meet the July 1, 2019 deadline, the Commission has no viable option other than removing those documents older than three years from its website.

At present, the Commissionís website allows a member of the public to find meeting notices, agendas, and briefing book materials, including attachments, going as far back as 2000. Commission staff has identified more than 1,000 documents totaling more than 40,000 pages that are less than three years old that the Department will remediate and migrate to the Commissionís new website by July 1, 2019. Other select documents, such as meeting agendas, vote lists, and program guidelines that are more than three years old, will also be remediated through a phased process. All other documents not identified for remediation and migration to the new website will be saved and provided upon request in accordance with the Commissionís Public Records Request and Records Retention Policies.
Logged
Daniel - California Highway Guy ● Highway Site: http://www.cahighways.org/ ●  Blog: http://blog.cahighways.org/ ● Follow California Highways on Facebook: https://www.facebook.com/cahighways

mrsman

  • *
  • Offline Offline

  • Posts: 2323
  • Age: 43
  • Location: Silver Spring, MD
  • Last Login: Today at 08:12:07 AM
Re: Caltrans Website Rework
« Reply #16 on: July 16, 2019, 08:11:52 AM »

Thank you Daniel for all your hard work on this matter.  Thank you in advance to anybody who will store some of this information for the public for the future.

I reiterate my earlier comments on this matter.  In my opinion it is shameful that the old material can't be stored as as historic matter exempt from the new requirements.  this will certainly have the effect of making it harder to access these public documents. It is what it is.

Nexus 5X

Logged

 


Opinions expressed here on belong solely to the poster and do not represent or reflect the opinions or beliefs of AARoads, its creators and/or associates.