fishing off a boardwalk

Me and my buddy have been fishing off the boardwalk in algonac, michigan for a couple of days now. We've been catching alot of small-medium perch and some rock bass. The one thing we've been wanting to catch but seem to not be able to is walleye. We have been using crawler harnesses with a 2 oz sinker about a foot down from that and just throwing it out there about 10 feet which is about 20 feet deep and letting it sit, to no prevail. What are we doing wrong if anyone has any input please chime in cause we really need help.