News:

Thank you for your patience during the Forum downtime while we upgraded the software. Welcome back and see this thread for some new features and other changes to the forum.

Main Menu

Caltrans Website Rework

Started by cahwyguy, July 06, 2019, 08:07:02 PM

Previous topic - Next topic

cahwyguy

I'm working on the highway pages this weekend, and I'm discovering that a lot of links to Caltrans documents -- bridge lots, EIRs, lists of projects approved by the CTC -- that were linked in articles or project pages have disappeared. They may being reworked, but that's poor form. Where I'm bumping into things, I'm tweeting to Caltrans so they can fix them (hopefully). I wanted to let others who might depend on Caltrans websites know, and encourage you to keep Caltrans on their toes and to keep the information out there.

Daniel
Daniel - California Highway Guy ● Highway Site: http://www.cahighways.org/ ●  Blog: http://blog.cahighways.org/ ● Podcast (CA Route by Route): http://caroutebyroute.org/ ● Follow California Highways on Facebook: https://www.facebook.com/cahighways


myosh_tino

There are a ton of "dead" links on the redesigned website.  Just about all of the links in the Traffic Ops section are currently broken and a quick sampling of the project documents links on the District 4 site are dead too.  With that said, a rework of this magnitude is no small undertaking and while I believe these documents (sign specs, project documents, etc) will reappear on the site, it's probably going to take some time.
Quote from: golden eagle
If I owned a dam and decided to donate it to charity, would I be giving a dam? I'm sure that might be a first because no one really gives a dam.

cahwyguy

We should probably keep tweeting and reporting the broken links.
Daniel - California Highway Guy ● Highway Site: http://www.cahighways.org/ ●  Blog: http://blog.cahighways.org/ ● Podcast (CA Route by Route): http://caroutebyroute.org/ ● Follow California Highways on Facebook: https://www.facebook.com/cahighways

Techknow

Around this time, Caltrans also seem to have removed most of the videos in their YouTube accounts (CaltransVideo and CaltransD7, perhaps more). I don't know if they're deleted, I know some of them are simply unlisted but they show up in the account's playlists.

nexus73

Your tax dollars NOT at work...LOL!

Rick
US 101 is THE backbone of the Pacific coast from Bandon OR to Willits CA.  Industry, tourism and local traffic would be gone or severely crippled without it being in functioning condition in BOTH states.

ClassicHasClass

Yes, it's a great look for them after the gas tax went up again.  :rolleyes:

fungus

It's related to AB 434 compliance. The environmental documents were all deleted too. https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=201720180AB434

cahwyguy

Interesting. You can be compliant and still make documents available, so hopefully those documents will return (as that transparancy, I recall, is required by other laws). We just need to keep pushing to bring the documents back.
Daniel - California Highway Guy ● Highway Site: http://www.cahighways.org/ ●  Blog: http://blog.cahighways.org/ ● Podcast (CA Route by Route): http://caroutebyroute.org/ ● Follow California Highways on Facebook: https://www.facebook.com/cahighways

J N Winkler

I didn't say anything here at the time since I don't customarily post about plans collection issues, but back in February I had to do a bottom-up redesign of the script I use to scrape plans and proposals for Caltrans advertised projects.  From about 2000 until (I think) late 2018/early 2019, Caltrans hosted the documentation for advertised contracts under a folder hierarchy on its main server and allowed directory navigation:

http://www.dot.ca.gov/hq/esc/oe/project_ads_addenda/

Each district had its own folder (zero-filled to two places) for its contracts, with subdirectories under each contract folder for plans, specs, addenda, and additional information (mainly CAD files and "informational handouts" that typically include geotechnical data).

Some time ago--I forget exactly when--Caltrans added language to the Office Engineer webpages indicating that a Caltrans account would be necessary to download plans, but did not start enforcing the requirement at that point.  Last winter they rejiggered the contract advertisement pages so that the /project_ads_addenda/ folders can no longer be navigated.  The folder structure has been preserved to an extent, but documents can now be downloaded only by using the partial directory path (scraped from webpages) as an URL argument to a webpage that will return a document only if a login cookie accompanies the HTTP request.

I am not happy about this change, but I have had to recognize that best-practice recommendations for Web design have advised against allowing directory navigation on a HTTP server for several years now, so servers that allow it have been getting rarer and rarer.
"It is necessary to spend a hundred lire now to save a thousand lire later."--Piero Puricelli, explaining the need for a first-class road system to Benito Mussolini

mrsman

Quote from: J N Winkler on July 11, 2019, 03:25:17 PM
I didn't say anything here at the time since I don't customarily post about plans collection issues, but back in February I had to do a bottom-up redesign of the script I use to scrape plans and proposals for Caltrans advertised projects.  From about 2000 until (I think) late 2018/early 2019, Caltrans hosted the documentation for advertised contracts under a folder hierarchy on its main server and allowed directory navigation:

http://www.dot.ca.gov/hq/esc/oe/project_ads_addenda/

Each district had its own folder (zero-filled to two places) for its contracts, with subdirectories under each contract folder for plans, specs, addenda, and additional information (mainly CAD files and "informational handouts" that typically include geotechnical data).

Some time ago--I forget exactly when--Caltrans added language to the Office Engineer webpages indicating that a Caltrans account would be necessary to download plans, but did not start enforcing the requirement at that point.  Last winter they rejiggered the contract advertisement pages so that the /project_ads_addenda/ folders can no longer be navigated.  The folder structure has been preserved to an extent, but documents can now be downloaded only by using the partial directory path (scraped from webpages) as an URL argument to a webpage that will return a document only if a login cookie accompanies the HTTP request.

I am not happy about this change, but I have had to recognize that best-practice recommendations for Web design have advised against allowing directory navigation on a HTTP server for several years now, so servers that allow it have been getting rarer and rarer.

Does this mean that you can no longer get the documents, or that it is a bit more cumbersome to do so?

cahwyguy

I'll note there's a similar directory approach for the underlying agenda items for the CTC minutes. I haven't done that part of my updates yet, so I have no ideas what horrors will be involved. For a while, they were embedding all the links in a gigantic PDF, which was a royal pain. But I'll find out in the next week or two when I finally get to doing the CTC minutes from April, May, and June. It looks like the CTC liaison page moved to https://dot.ca.gov/programs/transportation-programming/office-of-ctc-liaison-octcl , but the timed agenda page on that link isn't working. The CTC has a page at https://catc.ca.gov/ , and it does have a meetings and events page, but there's a runtime error on that page so you can't get any information. Time to go report it on Twitter.
Daniel - California Highway Guy ● Highway Site: http://www.cahighways.org/ ●  Blog: http://blog.cahighways.org/ ● Podcast (CA Route by Route): http://caroutebyroute.org/ ● Follow California Highways on Facebook: https://www.facebook.com/cahighways

J N Winkler

Quote from: mrsman on July 12, 2019, 09:02:02 AMDoes this mean that you can no longer get the documents, or that it is a bit more cumbersome to do so?

I still get the documents.  These changes forced me to sign up for a Caltrans account, and once I log in (which I can automate), everything downloads satisfactorily.  And because the URL arguments still preserve directory paths below contract folder level, I can still perform blockout, i.e., eliminating the URLs for previously downloaded documents from the download list so that only "fresh" stuff comes in each time the script is run.

What I have lost the ability to do is the following:

*  Go to a given district folder and navigate all the contract folders underneath, including folders for contracts that I missed when they were advertised (for a while, Caltrans was using the /project_ads_addenda/ hierarchy as a letting archive and re-uploading contracts that had been deleted from the server years earlier)

*  Go to a given contract folder and download all the documents listed, including "blind" ones that are not linked to from the advertising pages (because directory navigation is no longer possible, every document that is downloaded must have a corresponding partial URL path that can be scraped from a webpage)

Quote from: cahwyguy on July 12, 2019, 11:21:47 AMI'll note there's a similar directory approach for the underlying agenda items for the CTC minutes. I haven't done that part of my updates yet, so I have no ideas what horrors will be involved. For a while, they were embedding all the links in a gigantic PDF, which was a royal pain. But I'll find out in the next week or two when I finally get to doing the CTC minutes from April, May, and June. It looks like the CTC liaison page moved to https://dot.ca.gov/programs/transportation-programming/office-of-ctc-liaison-octcl , but the timed agenda page on that link isn't working. The CTC has a page at https://catc.ca.gov/ , and it does have a meetings and events page, but there's a runtime error on that page so you can't get any information. Time to go report it on Twitter.

At one time everything was navigable--EIRs, sign specs, . . . .  What a loss!

Regarding the big PDF, I suspect you may already be aware of this approach, but in a similar situation I would look to do PDF to plain text conversion (pdftotext command, IIRC part of the pdftools distribution, comes to mind) to try to grab the URLs using some combination of findstr and a Windows port of sed (if you are on Windows).  I have to do this with Indiana DOT pay item listings to identify which contracts have signing, and with Wisconsin DOT to match letting call numbers (typically of the format YYYYMMDDCCC) with eight-digit project numbers (which I use for filing signing sheets).  It works reasonably well, though there are wrinkles, such as dealing with random spaces that end up inside words as part of the conversion process.  I have not tried documents where the content I want is buried in the PDF code rather than being displayed explicitly as text--AIUI, extraction is still conceptually possible but involves decompressing PDF streams, which can be a hassle.  Pdftk has been my go-to for years, but I use a Windows port of GhostScript (useful for getting rid of unwanted features associated with PDF versions higher than 1.4) and I have found mutools (free) to be much more rapid for some tasks.

Of course, this all relies on getting hold of the documents in the first place, so I thank you for keeping after Caltrans on Twitter.
"It is necessary to spend a hundred lire now to save a thousand lire later."--Piero Puricelli, explaining the need for a first-class road system to Benito Mussolini

fungus

If Caltrans puts the construction plans behind passwords that eliminates the need to be AB 434 compliant. But EIRs have plans and technical documents and making a PDF W3 AA compliant could require alt text descriptions on basically every image, and OCRing the scanned in versions of public comment letters (and possibly retyping handwritten comments). https://www.w3.org/TR/WCAG20-TECHS/pdf#PDF7

cahwyguy

I'll note that I heard back from the CTC on this, and they noted: "As you can imagine the lift was very heavy getting everything remediated to acceptable ADA level. We sit under Caltrans web team and they and their vendors are working feverishly to complete the task for us and for themselves in addition to a couple other departments. We were told yesterday that 93% of our documents are remediated and the rest should done soon. They are currently working to link those items ."

Given what fungus noted, they could have quite a lift, given that many of the EIRs and public letters in the CTC minutes are scanned, and the PDF to character transition (I know because I often cut and paste) is not something that makes text to speech work well. Especially on older PDFs, that could be quite a bit of work. We often forget how hard accessibility is. I had a co-worker many many years ago at System Development Corporation who was a blind programmer, and I saw first hand the difficulties of screen to speech. It is much worse now in our very visual computer system world. (PS: She's the person who is behind the LA Radio Reading Service, http://www.larrs.org/ , a great organization)

Daniel - California Highway Guy ● Highway Site: http://www.cahighways.org/ ●  Blog: http://blog.cahighways.org/ ● Podcast (CA Route by Route): http://caroutebyroute.org/ ● Follow California Highways on Facebook: https://www.facebook.com/cahighways

mrsman

Quote from: fungus on July 12, 2019, 01:26:03 PM
If Caltrans puts the construction plans behind passwords that eliminates the need to be AB 434 compliant. But EIRs have plans and technical documents and making a PDF W3 AA compliant could require alt text descriptions on basically every image, and OCRing the scanned in versions of public comment letters (and possibly retyping handwritten comments). https://www.w3.org/TR/WCAG20-TECHS/pdf#PDF7

It is a shame that something that had good intentions has such bad consequences.  It is commendable to have good intentions to make all documentation accessible to those with disabilities, but it is highly impractical to be able to describe every technical detail so that blind people can understand it.  Unfortunately, it has been a long time since common sense prevailed in California.

The written comments should be made accessible though.  Even if it is laborious, it does provide information that would be useful to all, even the blind.  IMO, it shouldn't be too hard to hire readers to read the text alone - the same way they do for books on tape.

This whole saga is also reminiscent of throwing the baby out with the bath water.  Because the current documentation isn't compliant, they simply remove it all from public consumption.  It would have made a whole lot more sense to grandfather existing documentation and only put in the requirements for new letters.

AFAIK, the federal ADA requires new transit facilities to be accessible.  This generally means elevators for those who are wheelchair bound, but other improvements as well for other disabilities.  NYC has a lot of old subway stations without elevators.  The old stations have been grandfathered in.  But certain levels of improvements would require that they update the station for ADA purposes as well.  So there is a great financial disincentive to making necessary improvements.

More about this can be gleamed from this article:

https://www.reuters.com/article/us-new-york-subway-lawsuit/disability-rights-advocates-win-ruling-over-lack-of-nyc-subway-elevators-idUSKCN1QN2OC

Quote
NEW YORK (Reuters) - New York City's transit authority violated federal disability law when it replaced a subway station's stairs without installing an elevator, a federal judge has ruled, a decision that could require new elevators in future station renovations.

The ruling, issued on Tuesday by U.S. District Judge Edgardo Ramos in Manhattan, came as part of a lawsuit against the Metropolitan Transit Authority (MTA) filed in 2016 by disability rights groups, joined last year by the Justice Department.

Ramos found that when the MTA renovates a station in a way that affects its usability, such as by replacing stairs, the federal Americans with Disability Act requires it to install an elevator unless it is technically infeasible, regardless of cost.

cahwyguy

I found a contact at the CTC who provided me with the agenda item background for May and June. One item was one the website rework. This is from that item. Note item 3. What this means is that we should be requesting items more than 3 years old that we have identified (I already have a request in for the bridge logs). We should consider, either on my site or on AAroad or other sites, as appropriate, hosting these items as a public service as the state is not able to.

The Commission's Website Accessibility Project includes three primary tasks that must be completed before the Department's Chief Information Officer will certify compliance by July 1, 2019. These tasks are as follows:


  • The Commission's website must be redesigned and rebuilt using the most recently revised version of the State of California's Website Template to ensure the underlying infrastructure is compliant with accessibility standards.
  • All content migrated to the Commission's newly redesigned website (documents, images, graphics, videos, etc.) must be converted to meet the accessibility criteria established in AB 434. This complex conversion process is commonly referred to as document remediation.
  • The Commission must remove content older than three years from its website, and save it internally to be provided upon request.


Commission staff met with California State Transportation Agency (Agency) and Department staff on April 3, 2019, to discuss website compliance issues. At that time, Commission staff learned from Agency and the Department that documents more than three years old should be removed from the Commission's website since a) the documents are not or might not be compliant with the new standards, and (b) the cost to comply with legal accessibility requirements outweighs the public benefit of leaving older documents on the website.

The Commission staff appreciates that the Department is remediating the Commission's documents for the last three years to ensure the Commission is in compliance by July 1, 2019. Since the Department Chief Information Officer's signature is required for certification in addition to the signature of the Commission's Executive Director, and given the cost of outsourcing remediation of all documents currently on the Commission's website to meet the July 1, 2019 deadline, the Commission has no viable option other than removing those documents older than three years from its website.

At present, the Commission's website allows a member of the public to find meeting notices, agendas, and briefing book materials, including attachments, going as far back as 2000. Commission staff has identified more than 1,000 documents totaling more than 40,000 pages that are less than three years old that the Department will remediate and migrate to the Commission's new website by July 1, 2019. Other select documents, such as meeting agendas, vote lists, and program guidelines that are more than three years old, will also be remediated through a phased process. All other documents not identified for remediation and migration to the new website will be saved and provided upon request in accordance with the Commission's Public Records Request and Records Retention Policies.
Daniel - California Highway Guy ● Highway Site: http://www.cahighways.org/ ●  Blog: http://blog.cahighways.org/ ● Podcast (CA Route by Route): http://caroutebyroute.org/ ● Follow California Highways on Facebook: https://www.facebook.com/cahighways

mrsman

Thank you Daniel for all your hard work on this matter.  Thank you in advance to anybody who will store some of this information for the public for the future.

I reiterate my earlier comments on this matter.  In my opinion it is shameful that the old material can't be stored as as historic matter exempt from the new requirements.  this will certainly have the effect of making it harder to access these public documents. It is what it is.

Nexus 5X




Opinions expressed here on belong solely to the poster and do not represent or reflect the opinions or beliefs of AARoads, its creators and/or associates.