Multiple Asset sets with different validators

I would like to use two different asset sets on our site (both contain different metadata related to the upload).

I created a new one which I made the default for one of the folders within the asset folder and that works fine. Below these new folder only assetsets of the new type are created.

However there is also a requirement that the assetsets can contain files of differnent content types. The main one can contain all kinds of files (pdf, word etc) but the new one only mp4 files.

Is is possible to have a different configuration for the DefaultUploadValidationService for different assetsets? If so where, and how, do I configure this.