I am trying to saw vinyl purse with lining and the stitches are fine on the vinyl, but on the back of it are loose like in a knot. what's the problem?

Are you able to remove the bobbin case from the machine? Remove the needle and then work at gently removing any knots of thread in the bobbin area until you are able to remove the bobbin case.

Once the machine is cleaned of all the excess thread, lint, etc., install a BRAND NEW NEEDLE, reinstall the bobbin, and rethread the top thread. Then retest your machine on regular woven fabric. If it stitches, then you are probably okay. If it will not pick up the bobbin thread, then the machine is probably knocked out of time.

Search for: sewing machine timing

or take it for service.

FWIW, vinyl presents issues when sewing. A roller or teflon foot may help. Also, placing tissue paper between the presser foot and vinyl could be beneficial. Look at using a top-stitch needle to prevent the thread from hanging up on the vinyl. Search for web sites or videos on how to sew vinyl.

You shorten the stitch length and either reverse a few (5 or 6) stitches or stitch forward a few stitches. Some people will stitch in place but that frequently causes a knot on the underside of the fabric. You can also tie the thread tails together (especially when sewing fine fabric like chiffon, silk, etc.) and clip the tails close to the knot.

when you say knotting, do you mean a snarl up of looped thread on the underside. If so check top thread tension

no, you need the blade in place and cutting to give an even cut fabric edge. Set up for a three thread using the right hand needle, take out the left hand needle. move blade over to the right so it is cutting wide. Most overlockers there is also a thread finger you need to change on the foot, or a little lever you flick to move this finger forward into the stitching area. Check this on your manual as each make is a bit different.
If you can source it, put wooly nylon thread into the upper looper (knot it onto existing thread and just chain it through, the knot should go through the looper eye fine). Now turn the tension on top looper down so it is looser and tighten up tension on bottom looper, so needle 5, top looper 2 and bottom looper about 7.

Now test serge, the fabric cut edge should be wider than the overlocking seam so the fabric rolls under inside the stitching. Adjust the two looper threads so that this is happening, you want the bottom looper thread to nest right up against the needle stitch and the top looper thread to wrap all the way around too.

Once you've got this happening, turn the stitch length down to 0.5, to close the stitching right up. On some fabrics you'll need to adjust the differential feed too if the fabric is "waving" a bit.

Uneven stitches or sometimes loose when serging could indicate that one of the tension devices is faulting, so if this machine is not new and its doing this, it could need a service. Can you get a regular smooth three or 4 thread seam out if it???

It is always good to just recheck the threading path, make sure the thread aerial is up and that a thread hasn't got caught back on the thread stand somewhere if you're getting something wonky happening, threads stream off overlocker cones through the machine so anything wrong in the thread path will throw off the stitching.

Try adjusting the top tension dial to tighten up the loose stitching on the underside. Generally if you have loose threads underneath, its the top tension and visa versa.

However, if you are using a thread heavier than normal sewing thread through the needle and bobbin (which you probably need for this canvas) then the machine may struggle. I've sewed vinyl seat covers and gave up with my domestic machine and purchased a second hand industrial Singer machine 20U to cope with this heavier weight textile. Generally for anything heavier than upholstery fabric a domestic machine may not be strong enough.

my suggestion is to try different thread
check the tension
make sure there's no loose thread's caught up in the tension spring

Some sergers have a threading guide inside the door on the front of the machine.

If you need a manual, please provide the brand & model.

Try oiling the hook if you haven't recently.
Check you needle plate and hook for damage, they may need to be buffed or sanded smooth.
Check your upper tension discs for lint and double check your upper thread path (proper threading).
There are other things like hook timing, needle clearance or broken check spring that may be wrong but you need a technician for those things.
best of luck!

"Birdnesting" occurs when there is no tension on the needle thread to pull the bobbin thread. This is easily corrected by rethreading the machine, following the threading path carefully and making sure the thread is fully engaged in the tension mechanism.

The "knotting up" can reveal a lot. If you have loose threads on one side or the other, the tension on the opposite side will be the culprit.
So if the threads are on the underside as you sew, it is the top tension. Top tension ought to be between 4 & 6 (this variation to allow for the different weights of fabric in your projects).

If tensions appear correct, and the thread is definately in the channel between the discs, try raising presser foot, remove thread and with a strip piece of fabric 8 - 10" (20 - 25cm) gently insert the fabric strip and clean between the discs with a see saw / to and fro action.

It may be there is lint trapped between the discs, this will keep them slightly apart and reduce the actual tension, sometimes dramatically.

If the loose threads are on the top, it is bobbin tension that is loose, it too may have lint in the spring and be giving a "false" tension.
I would not recommend fiddling with bobbin tension without being shown, it may end up with missing small screws and spring pieces.

Hope this is of some help,
Bargainbox