Validation attributes not working on shared blocks when creating them

Description

Steps to reproduce:
1. Create ValidationAttribute and decorate a property on an editorial block for instance. Something like this:

[Display(GroupName = SystemTabNames.Content)]
[Required]
[ValidOrNot]
public virtual string test { get; set; }

[AttributeUsage(AttributeTargets.Property)]
public class ValidOrNot : ValidationAttribute
{
        protected override ValidationResult IsValid(object value, ValidationContext validationContext)
        {
            var contentData = validationContext.ObjectInstance as IContentData;
            //if (contentData == null)
            //{
            //    //This attribute only handles instances of IContentData. return ValidationResult.Success;
            //}
            //if (!contentData.Property[validationContext.MemberName].IsModified)
            //{
            //    return ValidationResult.Success;
            //}
            if (contentData == null)
            {
                return ValidationResult.Success;
            }
            else { return new ValidationResult("Should appear on creation of block"); }
        }
}

2. Create a new block of this type.
3. Publish the block.

Expected: Should get validation error.
Actual: Nothing really happens. It shows the save status, but it still looks like you can publish.

Note: You get the validation error on save after you've done some changes to the block.