Fishermen are growing frustrated with restrictions on the Imperial Beach Pier, which only allows them to fish in a designated fishing area.