Leaf margins are the edges of leaves. Different species have different types of edges. You’ve probably seen those fun crafting scissors that cut paper to have different shaped edges- this is a similar result. Not sure what brand of scissors God uses.
Scientists have designated different names for the different shaped margins and it helps with plant/tree ID if you know at least some of them. I’ve gotten by pretty well just knowing lobed, smooth (or entire) and serrate. There’s a lot of information below, but you don’t need to learn it all at once. Most important is to get into the practice of going outside into the real world and looking at real plants where they grow, looking at leaf margins and noticing they are different, and identifying a few of the most basic shapes.
I’ve included a lot of information here just because I’ve decided it’s time to up my game.
Reading the above chart from left to right- the first green leaf has a smooth, or ‘entire’ edge. Some leaves with smooth margins: magnolia, persimmon, dogwood.
The yellow/gold (purely arbitrary colour choice) is a wavy edge. Solanum pseudocapsicum L, sometimes called Jerusalem cherry or capsicum weed or about a dozen other regionally varied names, has wavy leaf margins, more commonly called undulate today. Look at a picture here: https://keyserver.lucidcentral.org/weeds/data/media/Html/solanum_pseudocapsicum.htm
Sinuate is similar to wavy or undulate, but the indentations are deeper, more like a pie-crust edge made by pushing thumb and index fingers around the pie-crust circumference. Witch-hazel and chestnut oak have sinuate leaves. http://woodyplantstutorial.nres.illinois.edu/margins/mar-sinuate.html
Next is lobed leaves, where the indentations are even deeper than dinuated. They can be fairly evenly lobed- symmetrical, more or less, like most oaks and maple trees. – https://www.thoughtco.com/lobed-hardwood-leaves-tree-leaf-key-1343478
Crenate or scalloped edged leaves include the small ground-ivy and the taller golden ragwort. http://identifythatplant.com/crenate-leaf-margins/ Geraniums and mock strawberries also have scalloped margins.
Cleft looks similar to lobed, except where the indentations of lobed look like they could have been made by fingers, cleft indentations are sharper, thinner, and look more like scissor cuts. Honestly, for most identification I’d just count cleft and lobed together.
Serrate, Dentate, and incised have different levels of cuts or intentations. The main thing is that lobed and scalloped leaves are curved or rounded, and the serrated, dentate, and incised leaves have sharper, teethed edges. You can see several varieties here.
These sites are helpful if you want more information and tools to help you sort out plants and leaves- but remember, they cannot be a substitute for getting outside and looking at real leaves:
http://woodyplantstutorial.nres.illinois.edu/margins/mar-incised.htmlame
http://bioimages.vanderbilt.edu/tree-key/simple-leaf-trees.htm