An OpenCV story

 Our chassis builder was determined to get OpenCV working on the latest Raspberry Pi platform for our entry, but it wasn't as straight forward as a quick download...................

Installing OpenCV on Raspberry Pi Zero 2 W

A tale with a moral

I have been interested in using OpenCV for years, but the driving force of need was never strong enough to overcome the obstacles. For example, my bomb disposal robot needed to detect the bomb. I scoured the Internet for instruction on installing OpenCV, but in the time it took to try and fail a few times I was able to implement Canny edge detection myself, and it worked fine for that purpose.

Then came Pi Wars 2022. Suddenly the goalposts had moved. Whereas the bomb disposal robot, which moved slowly in any case, was happy with two detections a second, I reckoned that our Pi Wars 2022 robot would need at least twenty, and preferably forty. So, back to the Internet.

If you type “install opencv on raspberry pi” into Google, you get 839,000 results, and I'm here to tell you that the most common single category is someone who has read how to do it and is passing it on without actually trying it themselves. Why do they do it? It just makes more crud to wade through. The next category is people who have tried it and got it to work on a very specific configuration, but don't explain fully what that configuration was, so you can't tell if it's applicable. One common thread is that a full install from scratch can take days and is fraught with peril. I asked on Discord for a prepared disk image without success. In total, over time, looking for shortcuts, I probably spent several days not installing OpenCV

Eventually I gave up on shortcuts, bit the bullet, and found a recipe for doing it the hard way. https://pimylifeup.com/raspberry-pi-opencv/ It was marked “beginner” which turned out to be a bit optimistic, but basically it worked!

A few prerequisites may be missing, so I added the following:

sudo apt-get install libatlas-base-dev

sudo apt-get install libjasper-dev

sudo apt-get install libqtgui4

sudo apt-get install libqt4-test

sudo apt-get install libgtk2.0-dev

sudo apt-get install pkg-config

sudo apt install libopenblas-base

sudo apt install libopenblas-dev

sudo apt install liblapacke-dev

If they are already there, running the install will do no harm.

Anyway: I now have OpenCV working on a variety of Pis in the sense that all the facilities in there that I think I need are present. I have made the disk images available on Google Drive if anyone wants to save effort, but I would encourage people to do it themselves.

Moral: sometimes the hard way is the best way.

Colin Walls
05/Dec/2021

 

P.S. The Pi Camera

Anyone used to SLRs will find the Pi camera hard to understand, particularly as it is on the end of a long chain of hardware and software, all trying to be helpful, but obscuring what is really happening, so …

There is no diaphragm, so you can't adjust the F stop. 'ISO' affects analogue gain and has no effect on raw sensor sensitivity. The focus is factory set (supposedly to 2m - infinity) and glued into place. There is no shutter, so light is hitting the sensor at all times. The sequence of zeroing, waiting, and reading pixel rows is roughly equivalent to a rolling shutter though, and the waiting period is roughly equivalent to shutter speed.

It is a video camera, there are no stills in the conventional sense. When you 'capture' an image what is really happening is that you get given the last video frame that the camera happened to produce. This means that you can't synchronise two cameras by 'capturing' an image at the same instant. This is why frame rate is crucial, because (assuming a 30fps rate, for example) any frame could be up to 1/30 second old when you get it, and the two cameras could be up to 1/30 second different in time of capture.

One thing common to nearly all digital cameras is that each pixel captures only one colour, so the true resolution is ¼ what you think it is, as other values are interpolated. This means that the ¼ 'maximum resolution' resolution has the special quality of matching the natural binning of the Bayer quartets in the sensor and involves a lot less processing. For the Pi camera V1 this 'sweet spot' is 1296 x 972. For the V2 it is 1640 x 1232

If you want to know more, read

https://picamera.readthedocs.io/en/release-1.13/fov.html

Always something new

 While we're all busy with different things, somehow we get new ideas for PiWars challenges. So we have a new Hungry Cattle hopper design, this time both a measure, and also a funnel to direct the feed to the trough.

This design is to have a single hopper dispensing a measured amount of feed only into the trough. The main hopper is 122 x 100 x 104mm and the cylinder, which can rotate through 360 degrees, is 62 x 100mm long.  

In addition to considering a new hopper, there's also the idea of a side funnel to direct the feed independent of the hopper position. 

With this addition, the current chassis can drive up alongside the trough, dispense the feed, and then drive on saving time. Using the central front hopper only, the chassis has to drive up to the trough, dispense the feed, and then manoeuvre away from the trough, all of which is extra navigation.

Thinking of a decoration for the arena, instead of just plain cardboard, or MDF, a colour scheme. 

This looks a bit more field like, as well as having the useful 250mm grid lines for laying out the challenges. Not sure about having muddy pools yet!

When it comes to thinking about a challenge, we can be very analytical, so here are a few diagrams ahead of our full apple picker design.

This is a chassis and attachment volume model showing the extents that a any given design can occupy.

Applying this to the apple tree, we can view the approach distances and develop a strategy.

From this we can then design the apple picker to fit the strategy, QED :) 

We did lots of clever software work as well, write up on that next time. 

When will we five meet again

 Lots of things happen when we get together, lots of ideas passed around and a few demonstrations. 

First up was the Hungry Cattle hopper and filler. This had been interrupted by a dodgy 3D printer hotend, but soon fixed and construction completed. 

The wooden bit is just for demonstration. It was duly loaded with the cheapest rice we could find and tested out. This is a video of a couple of runs, the first in real time, the second in slow motion.

Generally very happy with this, but improvements to be made. One big one is to automatically close the hopper after emptying, to prevent the accident had later, of pouring rice into an open hopper and subsequently all over the bench and floor, important for when we do the real run as well so that the robot is ready for refill as soon as possible. Another is to add indicator lights to show open and closed statuses, this may be just a physical switch changing coloured LEDs but giving feedback on status. The final change would be the use of a centralised hopper below the three currently planned so that the robot only has to position one feed outlet.

This layout would also help with detecting trough position to ensure that the robot releases when the hopper is above the trough. This is getting to be quite a big attachment!

Next up we had a look at the Nature's Bounty challenge. We had already had a look individually at a solution to this challenge but now was the opportunity to collectively look at the size of robot and the size of the tree.

A demo of a trough to hold collected apples in was examined and the ease, or otherwise of catching and storing apples soon became interesting. The apples themselves are also part of the problem, and for the time being in the project, they are polystyrene balls. They are lightweight, can be gripped from any angle and don't bounce much when dropped, so are easier to control. The plan is to colour these and attach them to the tree using steel washers and magnets. So here's a clutch of 'apples' in a trough and cue lots of dropping apples off the tree.

Out of this came a new solution, putting the trough underneath the robot.

Not sure if this will take off yet, but it does fix a few problems, and unloading might need a door and drawer at the rear!!!

The strategy of storing picked apples now having direction, the approach to actually picking the apples was looked at, also lots of fun, each of us pretending to be robots picking apples and how they would go about it.

Three obvious approaches to the tree were looked at as follows. 

This one doesn't take much imagination, drive up to the tree branches at 90 degrees and pick. Being at right-angle to the branch means the apples are in an easy position to pick, and the robot chassis can be aligned easily.

This second one goes in diagonally. This is much more difficult as it needs to approach the tree at an angle meaning more complex navigation, and the robot would need to be aligned to the tree axis. But it does allow for the possibility of picking two branches without relocating the robot chassis, which would make it faster.

Picking 'end on' to the tree branches lends itself to removing one dimension from the picker, that of moving left or right, which the other positions required, as well as in/out and up/down. It is difficult to achieve within the competitions dimensions rules but that's part of being inventive. Someone building a robot with mecanum wheels might even be able to drive in a circle, picking as they go.

No final design was agreed upon, someone taking away the task to draw up the picker configuration for next time. 

We had a quick demonstration of the StereoPi in operation mounted on the robot chassis, it looks good, but the demands of the chassis designer are high and they aren't satisfied. More on this another day.

We talked about the first challenge Shepherds Pi, but this is a real agility test for a robot, and we decided to leave it to another day when the chassis is a bit more advanced and we can experiment with moving around the arena in remote control, rather than just theorise about solutions. 

The fourth challenge, Farmyard Tours, got lots of ideas, and we agreed to go off and submit them as emails to gather them all together, it's unlikely that much will be said about these here for the time being unless we can't do them!!!!

Finally, a key piece of design was agreed upon, the attachment mounting. This is important
so we all know what we are designing to, and basically its four mounting holes on an 80mm grid on an 90mm plate.

The lugs on top aren't part of the specification, they're just there to aid mounting an attachment, in this case the Hungry Cattle hoppers.

And that's it for another week, the hopper controller has already been built on Arduino, but needs migrating to Pico, we're all getting back together to see the apple picker design after the next Raspberry Jam, and the next meeting will have a run through of the Shepherds Pi challenge, probably in slow motion as none of us are RC experts!!!

Not according to plan

 It's not plain sailing, just running through an idea from end to end. A story from our chassis maestro who's been lent a StereoPi to test

On trying to set up WiFi on the StereoPi I got an error message: "No wireless LAN interfaces found". So I went on the Internet and found dozens of different 'solutions', and tried several of them without success. Then I had a cup of coffee, and, while drinking it I had a thought. I checked the spec. of the StereoPi and discovered that this version just doesn't HAVE any wireless LAN interfaces. I plugged in an old WiFi dongle, and it worked instantly. Aaaaargh

But how did it do...?

Using openCV on StereoPi V1 (=CM3, +3B+) in stereo with minimal processing

1280x480  9fps

640x240   18fps

384x144   18fps

And the compute module 4 is available from Radio Spares, hang on, for 22/May/2022 

In other news the Hungry Cattle challenge test attachment was under construction, the first 3D print produced a hopper. 

Next was supposed to be the rotating valve part, but it turned out to be made of sponge, the reason being that the hot end on the 3D printer was getting very clogged. It is now an ex-hot end.

Heating and trying to disassemble was no better than doing it cold, it was completely clogged, the thread tight as well. The final, 'I will tear this apart' attempt in a vice just crushed the tube, it wasn't moving.

No spares, so I now await replacements, though what I'm going to do with the six that are coming in a pack I'm not sure, this has lasted me years. 

Chassis

 With four of us involved we're doing bits in parallel, but one blog, so here's some other progress on the chassis. I must take a picture for next time because this is being built as I write. 

This is the fundamental chassis, which is in two parts, primarily for the fourth challenge, The Farmyard Tours, where we think a bit of suspension might come in handy. Currently its a big wheel robot, getting it's speed from lower rotation motors that are easier to control, though we may switch to stepper motors in the future, it's not planned yet. We do have to keep in mind the 500mm/s average competition speed. 

The next picture is bulking the design out a bit, again subject to change, but taking shape.

The wheels are 3D printed and have now been covered in a rubber solution to give them grip.

Apples

 The Nature's Bounty challenge has apples, and what the apples should look like and how we get them off the 'tree' is of interest. 

I've posted a picture of the apple picker idea, but it is just an outline.

This bit is about that jaw on the picker and how it might work with different apple shapes.

Some thoughts on apples, spherical or otherwise, The pictures show two designs of apples for the PiWars 2022 Nature’s Bounty challenge, one spherical and one made up of intersecting circles, together with some apple picker ideas. The apples have a maximum diameter of 40mm, the minimum gap between apple and trunk for the topmost apple is 10mm.

 

A.     This is a spherical apple with plain grippers. The grippers hold the apple at only one position each forming an axle about which the apple can rotate. The grippers have to apply sufficient force to ensure that enough friction is generated at the touch pints to grip the apple. The apple can be held by any point on the gripper, reducing the need for accuracy.

B.     This is a spherical apple with parallel bar grippers. The apple is held at four points and cannot rotate. The force required to hold the apple is much less than A as there is some support for the apple provided by the lower gripper arm, but significant accuracy is required to position the gripper. The gripper requires less space to operate than plain grippers.

C.     This is a spherical apple with a finger gripper. The apple is held at 8 points and cannot rotate. The force required to hold the apple is much less than A as there is some support for the apple provided by the lower gripper arms. The apple can be gripped at multiple points reducing the need for accuracy.

D.    This is a spherical apple with a modified gripper. The apple is held over a continuous area and cannot rotate. Significant accuracy is required to position this gripper and the size of the gripper may be too large to be able to grip the top apple on the tree due to the gap between apple and tree trunk.

E.      This is an apple made of intersecting circles with plain grippers. The grippers hold the apple at only one position each forming an axle about which the apple can rotate. The grippers have to apply sufficient force to ensure that enough friction is generated at the touch points to grip the apple. The apple can be held by any point on the gripper, reducing the need for accuracy. The apple is lighter than the spherical apple so the gripper requires less force.

F.       This is an apple made up of intersecting circles and parallel bar gripper. The apple is held at four points and cannot rotate. The force required to hold the apple is much less as there is some support for the apple provided by the lower gripper arm, but significant accuracy is required to position the gripper. The gripper requires less space to operate than plain grippers.

G.     This is an apple made up of intersecting circles and finger gripper. The apple is held at 8 points and cannot rotate. The force required to hold the apple is much less as there is some support for the apple provided by the lower gripper arms, but significant accuracy is not required to position the grippe, as with F.

H.    This is an apple made up of intersecting circles and a plain gripper. The gripper can apply a force over a wide area, provided it is positioned with a small amount of accuracy, and while the apple can rotate, this may be limited by the position of the gripper. There is some increase in torque on the gripper due to holding the apple away from the centre of gravity.

I.          This is an apple made up of intersecting circles with plain grippers, but the apple is rotated 45 degrees. The apple is held at 4 points and cannot rotate. The apple can be held at any point on the gripper vertically, but the gripper must extend beyond the far edge of the apple.

J.       This is an apple made up of intersecting circles and a parallel bar gripper, the apple is rotated 45 degrees. The apple is held at 8 points and cannot rotate. Significant accuracy is required to ensure that the gripper locates both vertically and horizontally. The gripper requires less space to operate than plain grippers.

K.    This is an apple made up of intersecting circles and a finger gripper, the apple is rotated 45 degrees. The apple is held at 16 points and cannot rotate. Significant accuracy is not required as with J.

L.       This is an apple made up of intersecting circles and a modified gripper, the apple is rotated 45 degrees. The apple is held over a continuous area and cannot rotate. There is a degree of self-alignment in the gripper but does require some basic accuracy. The gripper may be too large to be used with the topmost apple, as with gripper C.

Hoppers

 More thinking, will have to start doing soon! 

A bit of design thought on the Hungry Cattle challenge. We needed at hopper to supply feed so the following has been sketched up. It's basically a box, the hopper, with a funnel and valve mechanism underneath, opened with a simple servo movement.

If we make one big hopper, 1litre, then we'd only need one and it would weigh 1kg, but to distribute the weight and make the size more manageable, we use three identical ones, positioned on the front of the robot. 

 That isn't the real chassis, just something to get an idea of how it will fit together. As a first design its not to bad but needs a lot more in the way of components. The hopper separating from the mechanism, so we can just make three and reuse them between designs. The valve and hopper mechanism will likely be remade several times, as they have to accommodate the servo and the attachments between the two. The valve also has to be reasonably tight fitting and still move. Next update on this will be pictures of a real one, maybe even a working video!

A first get together

 Lots of small activities done to clarify what we need and how we might move forward and eventually we all met up at a local Raspberry Jam in Exeter library which was a bit naughty as we sort of dominated it with PiWars. Separate meeting arranged for two weeks later to talk robots and nothing else. 

What have we done this week?

Collected cardboard for mock-ups.

We still need large cardboard sheets to make an arena, but that will be remade several times so a steady source needed, though we have an Ikea table wrapping in store.

A first mock-up of a tree.

This is a cardboard version as per the instructions, but we suspect we will make a more substantial version out of acrylic sheet when we are further down the line. These things have to be made so we have an idea of the actual challenges in front of us, not just a theoretical of paper based exercise.

Navigation is key and being in the Advanced category, it should be good. One idea is to create reference points in the arena which an imaging system can recognise and create positioning information based on that. The following are a sequence of pictures of a basic version, coloured lights hard wired to display a pattern in the arena wall.

Well it looks good, but needs lots of coding to use, and permission from the organisers!

We have also experimented in making apples, with a Lego attachment. 3D printing Lego compatible shapes isn't difficult, we have the dimensions, and it may make a good system if allowed. 

They look good, :) 

Getting the fruit off the tree is what the challenge is all about, so a quick mock-up picture of where we might go, drawn on the ubiquitous Tinkercad.

The 'eyes' on this mock-up may be those of a StereoPi imaging kit for Raspberry Pi. A previous winner has offered to loan us a kit to experiment with, it looks very good, but not so much just in the box!

This followed on from experiments with ESP32-CAM modules, an ESP32 microcontroller with a camera attachment, which work very well but not flexible enough for what we want to do, and would be a diversion away from Raspberry Pi. 

Another idea for a fruit picker is a chain and hook, we have a lot of ideas!...and Lego

Final bits for the week were getting a feel for how much 'feed' we have to provide for one of the challenges. This is just basic measurements, but to fulfil the challenge we think 1kg of feed in approximately 850ml of hopper capacity. Measurements of this can be a bit basic!