SSH DenyGroups

This option will provide possibility for management of access for Users that are connecting to your server from particular IP.

In this way it's possible to limit access on application level.
So far I know it's not possible to do this kind of user access filter on firewall level.

Location: “/etc/ssh/sshd_config”
I personally prefer to put this rules at end of the file to make it readable.

This keyword can be followed by a list of user name patterns, separated by spaces. Login is disallowed for user names that match one of the patterns. Only user names are valid; a numerical user ID is not recognized. By default, login is allowed for all users. If the pattern takes the form USER@HOST then USER and HOST are separately checked, restricting logins to particular users from particular hosts. The allow/deny directives are processed in the following order: DenyUsers, AllowUsers, DenyGroups, and finally AllowGroups. All of the specified user and group tests must succeed, before user is allowed to log in.

See PATTERNS in ssh_config(5) for more information on patterns.

You can put each user in to separate line

DenyGroups group1                 # Deny access for "group1" from ANY IP
DenyGroups group2                 # Deny acesss for "group2" from ANY IP

Or you can put all access rules in to one line

DenyGroups group1 group2

/etc/init.d/sshd restart         # For example at CentOS 6
systemctl restart sshd           # At OS with systemd for example CentOS 7

[root@SSH_SERVER ~]# adduser user1      # Create user1 account and group "user1"
[root@SSH_SERVER ~]# adduser user2      # Create user2 account and group "user2"
[root@SSH_SERVER ~]# passwd user1       # Set password for user1
...
[root@SSH_SERVER ~]# passwd user2       # Set password for user2
...

[root@SSH_SERVER ~]# vi /etc/ssh/sshd_config                                     # Put your rules at the end of this file
....
DenyGroups user1                 # Deny access for "user1" from ANY IP
DenyGroups user2                 # Deny acesss for "user2" from ANY IP

[root@SSH_SERVER ~]# systemctl restart sshd                                       # Restart SSHD so that the change will take efect

[user@SSH_CLIENT ~]$ ssh user1@SSH_SERVER            # Use rulle "DenyGroups user1"
user1@SSH_SERVER's password:
Permission denied, please try again.
user1@SSH_SERVER's password:
Permission denied, please try again.
user1@SSH_SERVER's password:
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[user1@SSH_SERVER ~]$ ssh user2@127.0.0.1            # Use rulle "DenyGroups user2"
root@127.0.0.1's password:
Permission denied, please try again.
root@127.0.0.1's password:
Permission denied, please try again.
root@127.0.0.1's password:
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).