Well, for those interested in the answer, I just came accross this package : EPiServer.Labs.Find.Toolbox

Here's the approach from this package (extracted from the readme) "Currently the synonym expansion is done in backend (Elastic Search) and relies on a synonym index. We solve this by retrieving and caching the synonym list and expand the matching synonyms on the query, on the client side."

Neat, it means I can use the .UsingSynonymsImproved() and change the SynonymLoader to also filter by siteid, something like so : 

public Dictionary<String, HashSet<String>> GetSynonyms(int synonymBatchSize, RuntimeCacheAdapter cache, StaticCachePolicy staticCachePolicy)
{
	var statisticLanguageTags = GetStatisticLanguageTags().ToList();
	var httpRequest = HttpContext.Current.GetRequestContext().HttpContext;
	var site = GetSiteDefinitionByRequest(httpRequest.Request.Url);
	statisticLanguageTags.Add($"siteid:{site.Id}");

	string synonymCacheKey = GetSynonymCacheKey(statisticLanguageTags);

	Dictionary<String, HashSet<String>> synonymsCached;
	if (TryGetCachedSynonym(synonymCacheKey, cache, out synonymsCached))
	{
		return synonymsCached;
	}

	var loadedSynonyms = LoadSynonymsFromSourceIndex(synonymBatchSize, statisticLanguageTags);
	var synonymDictionary = CreateSynonymDictionary(loadedSynonyms);

	cache.AddOrUpdate(synonymCacheKey, staticCachePolicy, synonymDictionary);
	return synonymDictionary;
}

Adding Synonym is super easy (_settingsService is a service layer I use to call SiteDefinition)

public ActionResult AddSynonym(string phrase, string synonym)
{
	var site = _settingsService.GetSiteDefinitionByRequest(Request.Url);
	var synonymInstance = new EPiServer.Find.Optimizations.Synonyms.Api.Synonym(phrase, synonym);

	synonymInstance.Tags = (synonymInstance.Tags ?? new List<string>()).Concat(new string[]
		{
			//"language:en", do not add manually, use the parameter on .Add() otherwise it will add as all languages
			$"siteid:{site.Id}"
		});

	var result = _findClient
		.Optimizations()
		.Synonyms()
		.Add(synonymInstance, Language.English);

	return RedirectToAction("Index");
}

That's all.