Another is culture penalty. I'm not exactly sure how it's calculated, but it's based on buildings from foreign cultures. There are six cultures - Barbarian (Britons, Dacia, Gauls, Germans, Scythia, Spain), Carthaginian (Carthage, Numidia), Eastern (Armenia, Parthia, Pontus), Egyptian (Egypt), Greek (Greek Cities, Macedon, Seleucid, Thrace), and Roman (Brutii, Julii, Scipii, Senate), and conquering within your culture should give you no penalty.
Government buildings and religious buildings having the biggest effect, so the first thing you want to do to reduce it is destroying any religious building they might have and building your own. Then upgrade all buildings you can, especially government building, as upgrading removes culture penalty for them. If you cannot upgrade, you can sometimes destroy and rebuild, what's going to cost you some money but it will work. Unfortunately you cannot destroy many buildings, especially the important government buildings, so you might need to live with some culture penalty forever.
The next important factor is unrest. Unrest starts high when you conquer a city, and then goes down by 5% until reaching the base level, but only if public order is at least 75%. So if you can get a conquered city to 75% by temporary measures like large garrison and good governor then just by staying in the city for a few turns the unrest will disappear. On the other hand if you cannot even reach 75% you will have pretty much permanent problems with the city. Well, not really permanent as rioting kills of citizens, what solves your problem slowly, but if you think the city is going to be a problem, just enslave it, and by the time it grows back it will have no unrest. Basic unrest rates are 0% for most cities but can be 15% for many and even 30% for a few like Jerusalem.
The next factor is squalor, based on city size. Big cities simply have high squalor, and it makes people unhappy. That's why when you're conquering a new city, with high distance to capital, high culture penalty, high unrest, and high squalor, it will be totally ungovernable. Better to enslave or even exterminate its population to reduce squalor, then quickly raze their temples and build your own to reduce culture penalty, keep large garrison in it for a few turns to reduce unrest, and keep capital close to center of your empire to make distance to capital penalties manageable.
To increase public order you can build buildings, especially religious buildings which usually give you +10% increase per level. Some give you only +5% and some other bonus, so you need to decide if the extra bonus is worth losing 5-25% of public order (depending on city size, unfortunately you cannot change your mind easily later, you'd need to raze the temple completely and start from shrine). Depending on culture there are some other buildings like execution square, odeon, tavern etc.
Wonders are also helpful here. Statue of Zeus at Olympia gives you permanent bonus to all your settlements, capturing any wonder gives you a high bonus for a few turns but it wears off too quickly to be worth relying on.
Influence rating of the governor improves public unrest, but you won't have enough governors with high influence for all of your empire. I never did it, but maybe you could try a ****ty governor and building an academy for him to get better stats. Governors can also get useful retinue from religious building.
A very important way of improving public order is garrison. The most important fact is that the bonus is based on ratio of total number of troops to total population. So 30 peasants are worth as much as 30 cataphracts. But everything including Town Guard is significantly more expensive per person than peasants - so seriously use only peasants as a garrison. Using serious combat troops for garrison in all your cities is going to cost you a lot of money, and you cannot even use them for crushing rebellions, as then you'd need another way of keeping your public order up.
Also - only use garrisons in small to medium settlements. In a small settlement, one stack of peasants can give you +80% garrison bonus (you can never go higher than that) and will cost you 100 denari/turn. In a large settlement, one stack of peasants will give you less than +5% and sitll cost you 100 denari/turn. Every other method costs the same no matter the settlement size. So it simply makes no sense to use garrison as a way of keeping large settlements happy, unless you positively have to. Much more effective ways are low taxes, religious buildings, governor influence, games, wonders etc. Oh and having absolutely no troops in a settlement gives it extra -15%, not to mention making it super easy to lose, so keep at least some peasants in all of them.