Here is a link to the ifixit guide on how to repair the auxiliary port in the speaker. It is handy because it details how to access the board on which the charge port is also located.
Here is an image, taken from the repair guide showing the port (highlighted in green)
(click on image to enlarge for better viewing)
As the charge port appears to be a SMD (surface mounted device) type mini USB connector you will need to have the expertise and tools necessary to unsolder the old port and solder on the new one. This procedure is not for a novice repairer, as it is twice as difficult in this instance as you are dealing with a double sided board and there is a standard SMD type USB port mounted directly underneath it. If you apply too much heat to remove the charge port you may end up accidentally removing the USB port attached below it as well.
I recommend that you contact a reputable, professional mobile phone repair or electronics repair service and ask for a quote to repair the port. That way even though it may cost you, they will be responsible for the repair and any damage (hopefully not if they are professional) that may occur.
Do not try to bypass the port and charge the battery directly as there most probably is battery management circuitry in the speaker to protect the battery (and the user) whilst it is being charged.