I've followed AsphaltPlanet's approach at many construction sites over the years and not had a problem. The 2013 Wichita road meet included a stop at the I-235/13th Street flyover, which was then under construction, and several of us inspected pier foundations without being bothered. This was in a suburban setting next to a flood control levee, with easy access from neighborhood streets.
I haven't checked how many construction contracts in the US include requirements to maintain perimeter security in their specifications, let alone include appurtenances such as site fencing as pay items. The general rule of thumb is that unless there is a clear justification in terms of liability or loss prevention, contractors don't want to shrink their profit margins by paying extra for site security. They will take it seriously, however, for projects in bad urban neighborhoods where there is a real risk of materials or equipment being stolen.
In the EU it may be different because construction contracts (presumably in response to an EU directive) invariably have a health-and-safety section in their documentation that addresses site safety, such as requirements for protective apparel.